Search Results: "Lucas Nussbaum"

6 October 2016

Reproducible builds folks: Reproducible Builds: week 75 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday September 25 and Saturday October 1 2016: Statistics For the first time, we reached 91% reproducible packages in our testing framework on testing/amd64 using a determistic build path. (This is what we recommend to make packages in Stretch reproducible.) For unstable/amd64, where we additionally test for reproducibility across different build paths we are at almost 76% again. IRC meetings We have a poll to set a time for a new regular IRC meeting. If you would like to attend, please input your available times and we will try to accommodate for you. There was a trial IRC meeting on Friday, 2016-09-31 1800 UTC. Unfortunately, we did not activate meetbot. Despite this participants consider the meeting a success as several topics where discussed (eg changes to IRC notifications of tests.r-b.o) and the meeting stayed within one our length. Upcoming events Reproduce and Verify Filesystems - Vincent Batts, Red Hat - Berlin (Germany), 5th October, 14:30 - 15:20 @ LinuxCon + ContainerCon Europe 2016. From Reproducible Debian builds to Reproducible OpenWrt, LEDE & coreboot - Holger "h01ger" Levsen and Alexander "lynxis" Couzens - Berlin (Germany), 13th October, 11:00 - 11:25 @ OpenWrt Summit 2016. Introduction to Reproducible Builds - Vagrant Cascadian will be presenting at the SeaGL.org Conference In Seattle (USA), November 11th-12th, 2016. Previous events GHC Determinism - Bartosz Nitka, Facebook - Nara (Japan), 24th September, ICPF 2016. Toolchain development and fixes Michael Meskes uploaded bsdmainutils/9.0.11 to unstable with a fix for #830259 based on Reiner Herrmann's patch. This fixed locale_dependent_symbol_order_by_lorder issue in the affected packages (freebsd-libs, mmh). devscripts/2.16.8 was uploaded to unstable. It includes a debrepro script by Antonio Terceiro which is similar in purpose to reprotest but more lightweight; specific to Debian packages and without support for virtual servers or configurable variations. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible in our testing framework after being fixed: The following updated packages appear to be reproducible now for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.) Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Reviews of unreproducible packages 77 package reviews have been added, 178 have been updated and 80 have been removed in this week, adding to our knowledge about identified issues. 6 issue types have been updated: Weekly QA work As part of reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development A new version of diffoscope 61 was uploaded to unstable by Chris Lamb. It included contributions from: Post-release there were further contributions from: reprotest development A new version of reprotest 0.3.2 was uploaded to unstable by Ximin Luo. It included contributions from: Post-release there were further contributions from: tests.reproducible-builds.org Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

3 October 2016

B lint R czey: Harden Debian with PIE and bindnow!

pie-bindnow-debian Shipping Position Independent Executables and using read-only Global Offset Table was already possible for packages but needed package maintainers to opt-in for each package (see Hardening wiki) using the pie and bindnow Dpkg hardening flags. Many critical packages enabled the extra flags but there are still way more left out according to Lintian hardening-no-bindnow and hardening-no-pie warnings. Now we can change that. We can make those hardening flags the default for every package.
We already have the needed patches for GCC (#835148) and dpkg (#835146, #835149). We already have all packages rebuilt once to test which breaks (Thanks to Lucas Nussbaum!). The Release Team already asked porters if they feel their ports ready for enabling PIE and most ports tentatively opted-in (Thanks to Niels Thykier for pushing this!). What is left is fixing the ~75 open bugs found during the test rebuilds and this is where You can help, too! Please check if your packages are affected or give a helping hand to other maintainers who need it. (See PIEByDefaultTransition wiki for hints on fixing the bugs.) Many thanks to those who already fixed their packages! If we can get past those last bugs we can enable those badly needed security features and make Stretch the most secure release ever!

7 September 2016

Reproducible builds folks: Reproducible Builds: week 71 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 28 and Saturday September 3 2016: Media coverage Antonio Terceiro blogged about testing build reprodubility with debrepro . GSoC and Outreachy updates The next round is being planned now: see their page with a timeline and participating organizations listing. Maybe you want to participate this time? Then please reach out to us as soon as possible! Packages reviewed and fixed, and bugs filed The following packages have addressed reproducibility issues in other packages: The following updated packages have become reproducible in our current test setup after being fixed: The following updated packages appear to be reproducible now, for reasons we were not able to figure out yet. (Relevant changelogs did not mention reproducible builds.) The following 4 packages were not changed, but have become reproducible due to changes in their build-dependencies: Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Reviews of unreproducible packages 706 package reviews have been added, 22 have been updated and 16 have been removed in this week, adding to our knowledge about identified issues. 5 issue types have been added: 1 issue type has been updated: Weekly QA work FTBFS bugs have been reported by: diffoscope development diffoscope development on the next version (60) continued in git, taking in contributions from: strip-nondeterminism development Mattia Rizzolo uploaded strip-nondeterminism 0.023-2~bpo8+1 to jessie-backports. A new version of strip-nondeterminism 0.024-1 was uploaded to unstable by Chris Lamb. It included contributions from: Holger added jobs on jenkins.debian.net to run testsuites on every commit. There is one job for the master branch and one for the other branches. disorderfs development Holger added jobs on jenkins.debian.net to run testsuites on every commit. There is one job for the master branch and one for the other branches. tests.reproducible-builds.org Debian: We now vary the GECOS records of the two build users. Thanks to Paul Wise for providing the patch. Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

5 September 2016

Raphaël Hertzog: My Free Software Activities in August 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. This months is rather light since I was away in vacation for two weeks. Kali related work The new pkg-security team is working full steam and I reviewed/sponsored many packages during the month: polenum, accheck, braa, t50, ncrack, websploit. I filed bug #834515 against sbuild since sbuild-createchroot was no longer usable for kali-rolling due to the embedded dash. That misfeature has been reverted and implemented through an explicit option. I brought the attention of ftpmasters on #832163 since we had unexpected packages in the standard section (they have been discovered in the Kali live ISO while we did not want them). I uploaded two fontconfig NMU to finally push to Debian a somewhat cleaner fix for the problem of various captions being displayed as squares after a font upgrade (see #828037 and #835142). I tested (twice) a live-build patch from Adrian Gibanel Lopez implementing EFI boot with grub and merged it into the official git repository (see #731709). I filed bug #835983 on python-pypdf2 since it has an invalid dependency forbidding co-installation with python-pypdf. I orphaned splint since its maintainer was missing in action (MIA) and immediately made a QA upload to fix the RC bug which kicked it out of testing (this package is a build dependency of a Kali package). django-jsonfield I wrote a patch to make python-django-jsonfield compatible with Django 1.10 (#828668) and I committed that patch in the upstream repository. Distro Tracker I made some changes to make the codebase compatible with Django 1.10 (and added Django 1.10 to the tox test matrix). I added a Debian Maintainer Dashboard link next to people s name on request of Lucas Nussbaum (#830548). I made a preliminary review of Paul Wise s patch to add multiarch hints (#833623) and improved the handling of the mailbot when it gets MIME Headers referencing an unknown charset (like cp-850 , Python only knows of cp850 ) I also helped Peter Palfrader to enabled a .onion address for tracker.debian.org, see onion.debian.org for the full list of services available over Tor. Misc stuff I updated my letsencrypt.sh salt formula to work with the latest version of letsencrypt.sh (0.2.0) I merged updated translations for the Debian Administrator s Handbook from weblate.org and uploaded a new version to Debian. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

13 July 2016

Niels Thykier: Selecting key packages via UDD

Thanks to Lucas Nussbaum, we now have a UDD script to filter/select key packages. Some example use cases: Which key packages used compat 4? 
# Data file compat-4-packages (one *source* package per line)
$ curl --silent --data-binary @compat-4-packages \
  https://udd.debian.org/cgi-bin/select-key-packages.cgi
alsamixergui
apg
[...]
sgml-base
wwwconfig-common
Also useful for things like bug#830997, which was my excuse for requesting this.:) Is package foo a key package (yet)? 
$ is-key-pkg()   
 RES=$(echo "$1"   curl --silent --data-binary @- \
    https://udd.debian.org/cgi-bin/select-key-packages.cgi)
 if [ "$RES" ]; then
   echo yes
 else
   echo no
 fi
 
$ is-key-pkg bash
yes
$ is-key-pkg mscgen
no
$ is-key-pkg NotAPackage
no
Above shell snippets might need tweaking for better error handling, etc. Once again, thanks to Lucas for the server-side UDD script.:)
Filed under: Debian

8 June 2016

Gunnar Wolf: University degrees and sysadmin skills

I'll tune in to the post-based conversation being held on Planet Debian: Russell Coker wonders about what's needed to get university graduates with enough skills for a sysadmin job, to which Lucas Nussbaum responds with his viewpoints. They present a very contrasting view of what's needed for students And for a good reason, I'd say: Lucas is an academician; I don't know for sure about Russell, but he seems to be a down-to-the-earth, dirty-handed, proficient sysadmin working on the field. They both contact newcomers to their fields, and will notice different shortcomings. I tend to side with Lucas' view. That does not come as a surprise, as I've been working for over 15 years in an university, and in the last few years I started walking from a mostly-operative sysadmin in an academic setting towards becoming an academician that spends most of his time sysadmining. Subtle but important distinction. I teach at the BSc level at UNAM, and am a Masters student at IPN (respectively, Mexico's largest and second-largest universities). And yes, the lack of sysadmin abilities in both is surprising. But so is a good understanding of programming. And I'm sure that, were I to dig into several different fields, I'd feel the same: Student formation is very basic at each of those fields. But I see that as natural. Of course, if I were to judge people as geneticists as they graduate from Biology, or were I to judge them as topologists as they graduate from Mathematics, or any other discipline in which I'm not an expert, I'd surely not know where to start Given I have about 20 years of professional life on my shoulders, I'm quite skewed as to what is basic for a computing professional. And of course, there are severe holes in my formation, in areas I never used. I know next to nothing of electronics, my mathematical basis is quite flaky, and I'm a poor excuse when talking about artificial intelligence. Where am I going with this? An university degree (BSc in English, would amount to "licenciatura" in Spanish) is not for specialization. It is to have a sufficiently broad panorama of the field, and all of the needed tools to start digging deeper and specializing either by yourself, working on a given field and learning its details as you go, or going through a postgraduate program (Specialization, Masters, Doctorate). Even most of my colleagues at the Masters in Engineering in Security and Information Technology lack of a good formation in fields I consider essential. However, what does information security mean? Many among them are working on legal implications of several laws that touch our field. Many other are working on authenticity issues in images, audios and other such media. Many other are trying to come up with mathematical ways to cheapen the enormous burden of crypto operations (say, "shaving" CPU cycles off a very large exponentiation). Others are designing autonomous learning mechanisms to characterize malware. Were I as a computing professional to start talking about their research, I'd surely reveal I know nothing about it and get laughed at. That's because I haven't specialized in those fields. University education should give a broad universal basis to enter a professional field. It should not focus on teaching tools or specific procedures (although some should surely be presented as examples or case studies). Although I'd surely be happy if my university's graduates were to know everything about administering a Debian system, that would be wrong for a university to aim at; I'd criticize it the same way I currently criticize programs that mix together university formation and industry certification as if they were related.

Lucas Nussbaum: Re: Sysadmin Skills and University Degrees

Russell Coker wrote about Sysadmin Skills and University Degrees. I couldn t agree more that a major deficiency in Computer Science degrees is the lack of sysadmin training. It seems like most sysadmins learned most of what they know from experience. It s very hard to recruit young engineers (freshly out of university) for sysadmin jobs, and the job interviews are often a bit depressing. Sysadmins jobs are also not very popular with this public, probably because university curriculums fail to emphasize what s exciting about those jobs. However, I think I disagree rather deeply with Russell s detailed analysis. First, Version Control. Well, I think that it s pretty well covered in university curriculums nowadays. From my point of view, teaching CS in Universit de Lorraine (France), mostly in Licence Professionnelle Administration de Syst mes, R seaux et Applications base de Logiciels Libres (warning: french), a BSc degree focusing on Linux systems administration, it s not usual to see student projects with a mandatory use of Git. And it doesn t seem to be a major problem for students (which always surprises me). However, I wouldn t rate Version Control as the most important thing that is required for a sysadmin. Similarly Dependencies and Backups are things that should be covered, but probably not as first class citizens. I think that there are several pillars in the typical sysadmin knowledge. First and foremost, sysadmins need a good understanding of the inner workings of an operating system. I sometimes feel that many Operating Systems Design courses are a bit too much focused on the Design side of things. Yes, it s useful to understand the low-level mechanisms, and be able to (mentally) recreate an OS from scratch. But it s also interesting to know how real systems are actually built, and what are the trade-off involved. I very much enjoyed reading Branden Gregg s Systems Performance: Enterprise and the Cloud because each chapter starts with a great overview of how things are in the real world, with a very good level of detail. Also, addressing OS design from the point of view of performance could be a way to turn those courses into something more attractive for students: many people like to measure, benchmark, optimize things, and it s quite easy to demonstrate how different designs, or different configurations, make a big difference in terms of performance in the context of OS design. It s possible to be a sysadmin and ignore, say, the existence of the VFS, but there s a large class of problems that you will never be able to solve. It can be a good trade-off for a curriculum (e.g. at the BSc level) to decide to ignore most of the low-level stuff, but it s important to be aware of it. Students also need to learn how to design a proper infrastructure (that meets requirements in terms of scalability, availability, security, and maybe elasticity). Yes, backups are important. But monitoring is, too. As well as high availability. In order to scale, it s important to be able to automatize stuff. Russell writes that Sysadmins need some programming skills, but that s mostly scripting and basic debugging. Well, when you design an infrastructure, or when you use configuration management tools such as Puppet, in some sense, you are programming, and in terms of needs to abstract things, it s actually similar to doing object-oriented programming, with similar choices (should I use that off-the-shelf puppet module, or re-develop my own? How should everything fit together?). Also, when debugging, it s often useful to be able to dig into code, understand what the developer was trying to do, and if the expected behavior actually matches what you are seeing. It often results in spending a lot of time to create a one-line fix, and it requires very advanced programming skills. Again, it s possible to be a sysadmin with only limited software development knowledge, but there s a large class of things that you are unlikely to address properly. I think that what makes sysadmins jobs both very interesting and very challenging is that they require a very wide range of knowledge. There s often the ability to learn about new stuff (much more than in software development jobs). Of course, the difficult question is where to draw the line. What is the sysadmin knowledge that every CS graduate should have, even in curriculums not targeting sysadmin jobs? What is the sysadmin knowledge for a sysadmin BSc degree? for a sysadmin MSc degree?

15 May 2016

Bits from Debian: What does it mean that ZFS is included in Debian?

Petter Reinholdtsen recently blogged about ZFS availability in Debian. Many people have worked hard on getting ZFS support available in Debian and we would like to thank everyone involved in getting to this point and explain what ZFS in Debian means. The landing of ZFS in the Debian archive was blocked for years due to licensing problems. Finally, the inclusion of ZFS was announced slightly more than a year ago, on April 2015 by the DPL at the time, Lucas Nussbaum who wrote "We received legal advice from Software Freedom Law Center about the inclusion of libdvdcss and ZFS in Debian, which should unblock the situation in both cases and enable us to ship them in Debian soon.". In January this year, the following DPL, Neil McGovern blogged with a lot of more details about the legal situation behind this and summarized it as "TLDR: It s going in contrib, as a source only dkms module." ZFS is not available exactly in Debian, since Debian is only what's included in the "main" section archive. What people really meant here is that ZFS code is now in included in "contrib" and it's available for users using DKMS. Many people also mixed this with Ubuntu now including ZFS. However, Debian and Ubuntu are not doing the same, Ubuntu is shipping directly pre-built kernel modules, something that is considered to be a GPL violation. As the Software Freedom Conservancy wrote "while licensed under an acceptable license for Debian's Free Software Guidelines, also has a default use that can cause licensing problems for downstream Debian users".

2 May 2016

Vincent Bernat: Pragmatic Debian packaging

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult1 and many people prefer to use less constrained tools2 like fpm or CheckInstall. However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:
  1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.
  2. Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.
  3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101 In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package. As stated in the introduction, we won t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project. As an example, we will package memcached, a distributed memory cache. There are four files to create:
  • debian/compat,
  • debian/changelog,
  • debian/control, and
  • debian/rules.
The first one is easy. Just put 9 in it: 
echo 9 > debian/compat
The second one has the following content: 
memcached (0-0) UNRELEASED; urgency=medium
  * Fake entry
 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200
The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won t influence the generated binary packages.

The control file debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them. 
Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system
The source package is called memcached. We have to use the same name as in debian/changelog. We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any. The Description field contains a short description of the binary package.

The build recipe The last mandatory file is debian/rules. It s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version3 without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm: 
DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=$ VERSION -0~$ DISTRIBUTION 0
TARBALL=memcached-$ VERSION .tar.gz
URL=http://www.memcached.org/files/$ TARBALL 
wget -N --progress=dot:mega $ URL 
tar --strip-components=1 -xf $ TARBALL 
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v $ PACKAGEVERSION  \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"
You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102 At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev
Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command4: 
mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control
You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies If the resulting package is installed on a freshly installed machine, it won t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends
The resulting package will contain the following information: 
$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb   grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:
  • System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.
  • Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).
  • systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.
Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V If you want to provide a System-V init script, have a look at /etc/init.d/skeleton on the most ancient distribution you want to target and adapt it5. Put the result in debian/memcached.init. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a /etc/default/memcached file. You can ship one by putting its content in debian/memcached.default.

Upstart Providing an Upstart job is similar: put it in debian/memcached.upstart. For example: 
description "memcached daemon"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon
script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script
When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example: 
[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
[Install]
WantedBy=multi-user.target
We reuse /etc/default/memcached even if it is not considered a good practice with systemd6. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag. You also need to add a build-dependency to dh-systemd in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd
And you need to modify the default rule in debian/rules: 
%:
    dh $@ --with systemd
The extra complexity is a bit unfortunate but systemd integration is not part of debhelper7. Without those additional modifications, the unit will get installed but you won t get a proper integration and the service won t be enabled on install or boot.

Dedicated user Many daemons don t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user8. Add a debian/memcached.postinst file with the following content: 
#!/bin/sh
set -e
case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac
#DEBHELPER#
exit 0
There is no cleanup of the user when the package is removed for two reasons:
  1. Less stuff to write.
  2. The user could still own some files.
The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends , adduser
The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff). You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103 It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture There are four steps to build a regular Debian package:
  1. debian/rules clean should clean the source tree to make it pristine.
  2. debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.
  3. debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.
  4. debian/rules binary will pack the different file trees into binary packages.
You don t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages: 
#!/usr/bin/make -f
%:
    dh $@
For each of the four targets described above, you can run dh with --no-act to see what it would do. For example: 
$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test
Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit magic . For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL. If one of the helpers do not do the right thing, you can replace it by using an override target: 
override_dh_auto_configure:
    ./configure --with-some-grog
Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options: 
override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog
This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. In the initial memcached example, we overrode all those magic targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@ --with systemd
override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The dh_auto_clean target is hijacked to download and setup the source tree9. We don t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install. To get a better sense of the difference, here is a diff: 
--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test
It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:
  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
  • files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,
Here is an example using wildcards for debian/memcached.docs: 
doc/*.txt
If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example: 
scripts/memcached-tool usr/bin
Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples The GitHub repository contains some additional examples. They all follow the same scheme:
  • dh_auto_clean is hijacked to download and setup the source tree
  • dh_gencontrol is modified to use a computed version
Notably, you ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.
  1. People may remember the time before debhelper 7.0.50 (circa 2009) where debian/rules was a daunting beast. However, nowaday, the boilerplate is quite reduced.
  2. The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don t.
  3. There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded.
  4. You also need to install devscripts and equivs package.
  5. It s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using start-stop-daemon and if it sources /lib/lsb/init-functions before considering it. If it seems to fit, you can install it yourself in debian/memcached/etc/init.d/. debhelper will ensure its proper integration.
  6. Instead, a user wanting to customize the options is expected to edit the unit with systemctl edit.
  7. See #822670
  8. The Debian Policy doesn t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like _memcached). Another common usage is to use Debian- as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in ps and top because of its length.
  9. We could call dh_auto_clean at the end of the target to let it invoke make clean. However, it is assumed that a fresh checkout is used before each build.

7 April 2016

Petter Reinholdtsen: One in two hundred Debian users using ZFS on Linux?

Just for fun I had a look at the popcon number of ZFS related packages in Debian, and was quite surprised with what I found. I use ZFS myself at home, but did not really expect many others to do so. But I might be wrong. According to the popcon results for spl-linux, there are 1019 Debian installations, or 0.53% of the population, with the package installed. As far as I know the only use of the spl-linux package is as a support library for ZFS on Linux, so I use it here as proxy for measuring the number of ZFS installation on Linux in Debian. In the kFreeBSD variant of Debian the ZFS feature is already available, and there the popcon results for zfsutils show 1625 Debian installations or 0.84% of the population. So I guess I am not alone in using ZFS on Debian. But even though the Debian project leader Lucas Nussbaum announced in April 2015 that the legal obstacles blocking ZFS on Debian were cleared, the package is still not in Debian. The package is again in the NEW queue. Several uploads have been rejected so far because the debian/copyright file was incomplete or wrong, but there is no reason to give up. The current status can be seen on the team status page, and the source code is available on Alioth. As I want ZFS to be included in next version of Debian to make sure my home server can function in the future using only official Debian packages, and the current blocker is to get the debian/copyright file accepted by the FTP masters in Debian, I decided a while back to try to help out the team. This was the background for my blog post about creating, updating and checking debian/copyright semi-automatically, and I used the techniques I explored there to try to find any errors in the copyright file. It is not very easy to check every one of the around 2000 files in the source package, but I hope we this time got it right. If you want to help out, check out the git source and try to find missing entries in the debian/copyright file.

2 January 2016

Niels Thykier: dput change-all-of-debian.changes

Lucas Nussbaum recently did a blog post called Debian is still changing . I found it a very welcome continuation of his previous blog post on the same topic. I find the graphs very interesting and was very happy to learn that he included relative graphs this time. Now I can with relatively ease say that 69% of all Debian packages are using a dh-style build (source). We have another 15% using classic debhelper, which means that at least 84% of all packages uses debhelper directly. Assuming all CDBS based packages rely on the debhelper class , we are at 99%! The latter is certainly an assumption, although I suspect it is probably pretty accurate[1]. Now, it is very cute to have world dominance , but that is not my primary interest in these numbers. Instead, we can use these numbers to determine that: Such as automatic dbgsym packages, indexable build-id from dbg(sym) packages via Packages files[2], and replacing maintscripts with ldconfig triggers. All of these changes happen to be changes that could be trivially deployed with very little risk and very high efficiency[3]. Notably, none of them required a compat bump (or a new debhelper tool). Of course, I do not intend to say that every change can (or should) be deployed via debhelper and much less into an existing dh_cmd -tool. Notably, dh_strip is reaching its breaking point for content. And if we were to require a compat bump for your change, we can now at least see the adoption rate via lintian. :) Nevertheless, it is nice to know that (politics aside) there is some agility in the Debian build system! :) [1] I would very much love to see numbers to (dis)prove my assumption about CDBS + debhelper. In fact, an absolute number of packages not using debhelper (indirectly) in Debian would be very intriguing. [2] New fields by default end up the Packages file. See e.g. the Packages.xz file on the debug mirror or your apt-cache via: 
apt-cache show mscgen-dbgsym   grep ^Build-Ids
The latter assumes that you have the debug mirror in your sources list. [3] Efficiency here being features people rarely override/disable.
Filed under: Debhelper, Debian

31 December 2015

Rapha&#235;l Hertzog: My Free Software Activities in December 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 21.25 hours on Debian LTS. During this time I worked on the following things: Distro Tracker I put a big focus on tracker.debian.org work this month. I completed the switch of the mail interface from packages.qa.debian.org to tracker.debian.org and I announced the change on debian-devel-announce. The changes resulted in a few problems that I quickly fixed (like #807073) and some other failures seen only by me and that were generated by weird spam messages (did you know that a subject can t have a newline character but that it can be encoded and folded over multiple lines?). Related to that I fixed some services so that they send their mails to tracker.debian.org directly instead of relying on the old emails (they get forwarded for now but it would be nice to be able to get rid of that forward). I updated (with the help of Lucas Nussbaum) the service that forwards the Launchpad bugs to the tracker, I sent a patch to update the @packages.debian.org aliases (not yet applied), I updated the configuration of all git commit notice scripts in the Alioth collab-maint and python-modules project (many remain to be done). I asked Ubuntu s Merge-O-Matic to use the new emails as well (see LP 1525497). DAK and the Debian BTS still have to be updated, as of yet nobody reacted to my announce last but not least I updated many wiki pages which duplicated the instructions to setup the commit notice sent to the PTS. While on a good track I opted to tackle the long-standing RC bug that was plaguing tracker.debian.org (#789183), so I updated the codebase to rely on Twitter s bootstrap v4 instead of v2. I had to switch to something else for the icons since glyphicons is no longer provided as part of bootstrap and the actual license for the standalone version was not suitable for use. I opted for Github s Octicons. I made numerous little improvements while doing that (closing some bugs in the process) and I believe that the result is more pleasant to use. I also did a lot of bug triage and fixed a few small issues like the incomplete architecture list (#793547), or fixing a page used only by people with javascript disabled that was not working. Or the invalid links for packages still using CVS (ugh, see #561228). Misc packaging Django. After having added DEP-8 tests (as part of my LTS work, see above), I discovered that the current version in unstable did not pass its test suite so I filed the issue upstream (ticket 26016) and added the corresponding patch. And I encouraged others to update python-bcrypt in Debian to a newer version that would have worked with Django 1.9 (see #803096). I also fixed another small issue in Django (see ticket 26017 with my pull request that got accepted). I asked the release managers to consider accepting the latest 1.7.x version in jessie (see #807654) but I have gotten zero answer so far. And I m not the only one waiting an answer. It s a bit of a sad situation we still have a few weeks until the next point release but for once I do it in advance and I would love to have timely feedback. Last but not least, I started the maintaining the current LTS release (1.8.x) in jessie-backports. Tryton. I upgraded to Tryton 3.8 and discovered an issue that I filed in #806781. I sponsored 5 new tryton modules for Matthias Behrle (who is DM) as well as one security upload (for CVE-2015-0861). Debian Handbook. I uploaded a new version to Debian Unstable and requested (to the release managers) the permission to upload a backport of it to jessie so that jessie has a version of the package that documents jessie and not wheezy contrary to my other Django request, this one should be non-controversial but I also have had zero answer so far, see #807515. Misc. I filed #808583 when sbuild stopped working with Perl 5.22. I handled #807860 on publican, I found the corresponding upstream ticket and discovered a work around with the help of upstream (see here). Kali related work I reported a bug to #debian-apt about apt miscalculating download size (ending up with 18 EB!) which resulted in a fix here in version 1.1.4. Installing a meta-package that needed more than 2GB was no longer possible without this fix and we have a kali-linux-all metapackage in that situation that gets regularly installed in a Jenkins test. I added captcha support to Distro Tracker and enabled this feature on pkg.kali.org. I filed #808863 against uhd-host because it was not possible to install the package in a systemd-nspawn s managed chroot where /proc is read-only. And we started using this to test dist-upgrade from one version of Kali to the next Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

26 December 2015

Lucas Nussbaum: Debian packages with /outdated/ packaging style

(This is just a copy of this debian-devel@ email) Following my blog post yesterday with graphs about Debian packaging evolution, I prepared lists of packages for each kind of outdatedness. Of course not all practices highlighted below are deprecated, and there are good reasons to continue to do some of them. But still, given that they all represent a clear minority of packages, I thought that it would be useful to list the related packages. (I honestly didn t know if some of my packages would show up in the lists!) The lists are available at https://people.debian.org/~lucas/qa-20151226/ I also pushed them to alioth, so you can either do:
ssh people.debian.org 'grep -A 10 YOURNAME ~lucas/public_html/qa-20151226/*ddlist'
or:
ssh alioth.debian.org 'grep -A 10 YOURNAME ~lucas/qa-20151226/*ddlist' the meaning of the lists is: If you don t understand why your package is listed, you can have a look at allpackages-20151226.yaml that provides more details. If you still don t understand, just ask me. Excluding duplicates, a total of 5469 packages are listed. The dd-list output for the merged list is also available (which isn t very useful, except to know if you are listed).

25 December 2015

Lucas Nussbaum: Debian is still changing

Here is an update to the usual graphs generated from snapshot.d.o. See my previous blog post for the background info. In all graphs, it s easy to see the effect of the Jessie freeze (and the previous freezes since 2005, too). Team maintenance comaint-2015 It s interesting to see that, while the number of team-maintained packages increases, the number of packages that aren t co-maintained is very stable. Maintenance using a VCS vcs-2015 Git is the clear winner now, with the migration rate increasing recently. Packaging helpers helpers-2015 As usual, the number of packages using CDBS is quite stable. The number of packages using traditional debhelper might soon be lower than those using CDBS. Patch systems and packaging formats formats-patches-2015 3.0 is the clear winner, even if we still have 3000+ packages using 1.0, and ~1000 of those modifying files directly. The other patch systems have basically disappeared.
 So, all those graphs are kind-of boring now. Any good ideas of additional things to track, that be can identified reliably by looking at source packages? For those interested, below are links to the graphs with percentages of packages. comaint-percent-2015 formats-patches-percent-2015 vcs-percent-2015 helpers-percent-2015

28 August 2015

Lucas Nussbaum: DebConf 15

I attended DebConf 15 last week. After being on semi-vacation from Debian for the last few months, recovering after the end of my second DPL term, it was great to be active again, talk to many people, and go back to doing technical work. Unfortunately, I caught the debbug quite early in the week, so I was not able to make it as intense as I wanted, but it was great nevertheless. I still managed to do quite a lot: DC15 was a great DebConf, probably one of the two bests I ve attended so far. I m now looking forward to DC16 in Cape Town!

7 August 2015

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken. 
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings: 
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial: 
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug #770135). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug #792045. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken. 
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings: 
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial: 
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

20 June 2015

Lunar: Reproducible builds: week 4 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Lunar rebased our custom dpkg on the new release, removing a now unneeded patch identified by Guillem Jover. An extra sort in the buildinfo generator prevented a stable order and was quickly fixed once identified. Mattia Rizzolo also rebased our custom debhelper on the latest release. Packages fixed The following 30 packages became reproducible due to changes in their build dependencies: animal-sniffer, asciidoctor, autodock-vina, camping, cookie-monster, downthemall, flashblock, gamera, httpcomponents-core, https-finder, icedove-l10n, istack-commons, jdeb, libmodule-build-perl, libur-perl, livehttpheaders, maven-dependency-plugin, maven-ejb-plugin, mozilla-noscript, nosquint, requestpolicy, ruby-benchmark-ips, ruby-benchmark-suite, ruby-expression-parser, ruby-github-markup, ruby-http-connection, ruby-settingslogic, ruby-uuidtools, webkit2gtk, wot. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Also, the following bugs have been reported: reproducible.debian.net Holger Levsen made several small bug fixes and a few more visible changes: strip-nondeterminism Version 0.007-1 of strip-nondeterminism the tool to post-process various file formats to normalize them has been uploaded by Holger Levsen. Version 0.006-1 was already in the reproducible repository, the new version mainly improve the detection of Maven's pom.properties files. debbindiff development At the request of Emmanuel Bourg, Reiner Herrmann added a comparator for Java .class files. Documentation update Christoph Berg created a new page for the timestamps in manpages created by Doxygen. Package reviews 93 obsolete reviews have been removed, 76 added and 43 updated this week. New identified issues: timestamps in manpages generated by Doxygen, modification time differences in files extracted by unzip, tstamp task used in Ant build.xml, timestamps in documentation generated by ASDocGen. The description for build id related issues has been clarified. Meetings Holger Levsen announced a first meeting on Wednesday, June 3rd, 2015, 19:00 UTC. The agenda is amendable on the wiki. Misc. Lunar worked on a proof-of-concept script to import the build environment found in .buildinfo files to UDD. Lucas Nussbaum has positively reviewed the proposed schema. Holger Levsen cleaned up various experimental toolchain repositories, marking merged brances as such.

13 May 2015

Lucas Nussbaum: systemd: Type=simple and avoiding forking considered harmful?

(This came up in a discussion on debian-user-french@l.d.o) When converting from sysvinit scripts to systemd init files, the default practice seems to be to start services without forking, and to use Type=simple in the service description. What Type=simple does is, well, simple. from systemd.service(5):

If set to simple (the default value if neither Type= nor BusName= are specified), it is expected that the process configured with ExecStart= is the main process of the service. In this mode, if the process offers functionality to other processes on the system, its communication channels should be installed before the daemon is started up (e.g. sockets set up by systemd, via socket activation), as systemd will immediately proceed starting follow-up units. In other words, systemd just runs the command described in ExecStart=, and it s done: it considers the service is started. Unfortunately, this causes a regression compared to the sysvinit behaviour, as described in #778913: if there s a configuration error, the process will start and exit almost immediately. But from systemd s point-of-view, the service will have been started successfully, and the error only shows in the logs: 

root@debian:~# systemctl start ssh
root@debian:~# echo $?
0
root@debian:~# systemctl status ssh
  ssh.service - OpenBSD Secure Shell server
 Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
 Active: failed (Result: start-limit) since mer. 2015-05-13 09:32:16 CEST; 7s ago
 Process: 2522 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 2522 (code=exited, status=255)
mai 13 09:32:16 debian systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
mai 13 09:32:16 debian systemd[1]: Unit ssh.service entered failed state.
mai 13 09:32:16 debian systemd[1]: ssh.service start request repeated too quickly, refusing to start.
mai 13 09:32:16 debian systemd[1]: Failed to start OpenBSD Secure Shell server.
mai 13 09:32:16 debian systemd[1]: Unit ssh.service entered failed state.
With sysvinit, this error is detected before the fork(), so it shows during startup: 
root@debian:~# service ssh start
 [....] Starting OpenBSD Secure Shell server: sshd/etc/ssh/sshd_config: line 4: Bad configuration option: blah
 /etc/ssh/sshd_config: terminating, 1 bad configuration options
 failed!
 root@debian:~#
It s not trivial to fix that. The implicit behaviour of sysvinit is that fork() sort-of signals the end of service initialization. The systemd way to do that would be to use Type=notify, and have the service signals that it s ready using systemd-notify(1) or sd_notify(3) (or to use socket activation, but that s another story). However that requires changes to the service. Returning to the sysvinit behaviour by using Type=forking would help, but is not really a solution: but what if some of the initialization happens *after* the fork? This is actually the case for sshd, where the socket is bound after the fork (see strace -f -e trace=process,network /usr/sbin/sshd), so if another process is listening on port 22 and preventing sshd to successfully start, it would not be detected. I wonder if systemd shouldn t do more to detect problems during services initialization, as the transition to proper notification using sd_notify will likely take some time. A possibility would be to wait 100 or 200ms after the start to ensure that the service doesn t exit almost immediately. But that s not really a solution for several obvious reasons. A more hackish, but still less dirty solution could be to poll the state of processes inside the cgroup, and assume that the service is started only when all processes are sleeping. Still, that wouldn t be entirely satisfying

18 April 2015

Neil McGovern: Taking office

Yesterday, my first term started as the Debian Project Leader. There s been a large number of emails congratulating me, and thanks to everyone who sent those. I d also like to thank Mehdi Dogguy and Gergely Nagy for running, and of course Lucas Nussbaum for his service over the past two years. Lucas also did a great handover, and so (I hope!) I m aware of most of the issues that are ongoing. As started previously, I ll keep my daily log of activities in /srv/leader/news/ on master.debian.org.

Next.

Previous.